Heterogeneous Treatment Effects Research Articles

Navigating Potential Pitfalls in Difference-in-Differences Designs: Reconciling Conflicting Findings on Mass Shootings’ Effect on Electoral Outcomes

Work on the electoral effects of gun violence in the U.S. relying on difference-in-differences designs has produced findings ranging from null to substantively large effects. However, as difference-in-difference designs, on which this research relies, have exploded in popularity, scholars have documented several methodological issues including potential violations of parallel-trends and unaccounted for treatment effect heterogeneity. These pitfalls (and their solutions) have not been fully explored in political science. We apply these advancements to the unresolved debate on gun violence’s effects on U.S. electoral outcomes. We show that studies finding a large positive effect of gun violence on Democratic vote shares are a product of a failure to properly specify difference-in-differences models when underlying assumptions are unlikely to hold. Once these biases are corrected, shootings show little evidence of sparking large electoral change. Our work clarifies an unresolved debate and provides a cautionary guide for scholars currently employing difference-in-differences designs.

Policy Targeting under Network Interference

Abstract This paper studies the problem of optimally allocating treatments in the presence of spillover effects, using information from a (quasi-)experiment. I introduce a method that maximizes the sample analog of average social welfare when spillovers occur. I construct semi-parametric welfare estimators with known and unknown propensity scores and cast the optimization problem into a mixed-integer linear program, which can be solved using off-the-shelf algorithms. I derive a strong set of guarantees on regret, i.e., the difference between the maximum attainable welfare and the welfare evaluated at the estimated policy. The proposed method presents attractive features for applications: (i) it does not require network information of the target population; (ii) it exploits heterogeneity in treatment effects for targeting individuals; (iii) it does not rely on the correct specification of a particular structural model; and (iv) it accommodates constraints on the policy function. An application for targeting information on social networks illustrates the advantages of the method.

Disentangling Person-Dependent and Item-Dependent Causal Effects: Applications of Item Response Theory to the Estimation of Treatment Effect Heterogeneity

Analyzing heterogeneous treatment effects (HTEs) plays a crucial role in understanding the impacts of educational interventions. A standard practice for HTE analysis is to examine interactions between treatment status and preintervention participant characteristics, such as pretest scores, to identify how different groups respond to treatment. This study demonstrates that the identical patterns of HTE on test score outcomes can emerge either from variation in treatment effects due to a preintervention participant characteristic or from correlations between treatment effects and item easiness parameters. We demonstrate analytically and through simulation that these two scenarios cannot be distinguished if analysis is based on summary scores alone. We then describe a novel approach that identifies the relevant data-generating process by leveraging item-level data. We apply our approach to a randomized trial of a reading intervention in second grade and show that any apparent HTE by pretest ability is driven by the correlation between treatment effect size and item easiness. Our results highlight the potential of employing measurement principles in causal analysis, beyond their common use in test construction.

Heterogeneous treatment effects and optimal targeting policy evaluation

Heterogeneous treatment effects and optimal targeting policy evaluation

The incidence and effects of decentralized wage bargaining in Finland

AbstractThere was a strong push from employers to decentralize wage setting in Finland in the early 2000s. We analyze the incidence of decentralization and its effect on the level and dispersion of wages by using nationally representative panel data. The results show that wage setting was more likely decentralized in collective agreements where a high share of employees worked in manufacturing or real estate industries than in other industries, such as in education and human health and social work activities. Decentralization was, for the most part, quite short-lived. Using recent difference-in-differences methods that allow for heterogeneous treatment effects and differences in the timing of treatment, we show that decentralization had modest positive effects on the level and dispersion of wages in manufacturing.

Assessing treatment effect heterogeneity in the presence of missing effect modifier data in cluster-randomized trials.

Understanding whether and how treatment effects vary across subgroups is crucial to inform clinical practice and recommendations. Accordingly, the assessment of heterogeneous treatment effects based on pre-specified potential effect modifiers has become a common goal in modern randomized trials. However, when one or more potential effect modifiers are missing, complete-case analysis may lead to bias and under-coverage. While statistical methods for handling missing data have been proposed and compared for individually randomized trials with missing effect modifier data, few guidelines exist for the cluster-randomized setting, where intracluster correlations in the effect modifiers, outcomes, or even missingness mechanisms may introduce further threats to accurate assessment of heterogeneous treatment effect. In this article, the performance of several missing data methods are compared through a simulation study of cluster-randomized trials with continuous outcome and missing binary effect modifier data, and further illustrated using real data from the Work, Family, and Health Study. Our results suggest that multilevel multiple imputation and Bayesian multilevel multiple imputation have better performance than other available methods, and that Bayesian multilevel multiple imputation has lower bias and closer to nominal coverage than standard multilevel multiple imputation when there are model specification or compatibility issues.

318 Discovering Subgroups with Supervised Machine Learning Models for Heterogeneity of Treatment Effect Analysis

OBJECTIVES/GOALS: The goal of the study is to provide insights into the use of machine learning methods as a means to predict heterogeneity of treatment effect (HTE) in participants of randomized clinical trials. METHODS/STUDY POPULATION: Using data from 2,441 participants enrolled in the ASPirin in Reducing Events in the Elderly (ASPREE) randomized controlled trial of daily low-dose aspirin vs placebo in the United States, we developed multivariable risk prediction models for the composite outcome of dementia, disability, or death. We used two machine learning techniques, decision trees and random forests, to develop novel non-parametric outcomes classifiers and generate risk-based subgroups. The comparator method was an extant semi-parametric proportional hazards predictive risk model. We then assessed HTE by examining the 5-year absolute risk reduction (ARR) of aspirin vs placebo in each risk subgroup. RESULTS/ANTICIPATED RESULTS: In the random forest classifier, the ARR at 5 years in the highest risk quintile was 13.7% (95% CI 3.1% to 24.4%). For the semi-parametric proportional hazards model, the ARR in the highest risk quintile was 15.1% (95% CI 4.0% to 26.3%). These results were comparable and provide evidence of the viability of internally developed parsimonious non-parametric machine learning models for HTE analysis. The decision tree model results (5-year ARR = 17.0%, 95% CI= -5.4% to 39.4% in the highest risk subgroup) exhibited more uncertainty in the results. DISCUSSION/SIGNIFICANCE: None of the models detected significant HTE on the relative scale; there was substantial HTE on the absolute scale in three of the models. Treatment benefit on the absolute scale may be regarded as bearing greater clinical importance and may be present even in the absence of benefit on the relative scale.

Semaglutide versus placebo in people with obesity-related heart failure with preserved ejection fraction: a pooled analysis of the STEP-HFpEF and STEP-HFpEF DM randomised trials.

In the STEP-HFpEF (NCT04788511) and STEP-HFpEF DM (NCT04916470) trials, the GLP-1 receptor agonist semaglutide improved symptoms, physical limitations, bodyweight, and exercise function in people with obesity-related heart failure with preserved ejection fraction. In this prespecified pooled analysis of the STEP-HFpEF and STEP-HFpEF DM trials, we aimed to provide a more definitive assessment of the effects of semaglutide across a range of outcomes and to test whether these effects were consistent across key patient subgroups. We conducted a prespecified pooled analysis of individual patient data from STEP-HFpEF and STEP-HFpEF DM, randomised, double-blind, placebo-controlled trials at 129 clinical research sites in 18 countries. In both trials, eligible participants were aged 18 years or older, had heart failure with a left ventricular ejection fraction of at least 45%, a BMI of at least 30 kg/m2, New York Heart Association class II-IV symptoms, and a Kansas City Cardiomyopathy Questionnaire Clinical Summary Score (KCCQ-CSS; a measure of heart failure-related symptoms and physical limitations) of less than 90 points. In STEP-HFpEF, people with diabetes or glycated haemoglobin A1c concentrations of at least 6·5% were excluded, whereas for inclusion in STEP-HFpEF DM participants had to have been diagnosed with type 2 diabetes at least 90 days before screening and to have an HbA1c of 10% or lower. In both trials, participants were randomly assigned to either 2·4 mg semaglutide once weekly or matched placebo for 52 weeks. The dual primary endpoints were change from baseline to week 52 in KCCQ-CSS and bodyweight in all randomly assigned participants. Confirmatory secondary endpoints included change from baseline to week 52 in 6-min walk distance, a hierarchical composite endpoint (all-cause death, heart failure events, and differences in changes in KCCQ-CSS and 6-min walk distance); and C-reactive protein (CRP) concentrations. Heterogeneity in treatment effects was assessed across subgroups of interest. We assessed safety in all participants who received at least one dose of study drug. Between March 19, 2021 and March 9, 2022, 529 people were randomly assigned in STEP-HFpEF, and between June 27, 2021 and Sept 2, 2022, 616 were randomly assigned in STEP-HFpEF DM. Overall, 1145 were included in our pooled analysis, 573 in the semaglutide group and 572 in the placebo group. Improvements in KCCQ-CSS and reductions in bodyweight between baseline and week 52 were significantly greater in the semaglutide group than in the placebo group (mean between-group difference for the change from baseline to week 52 in KCCQ-CSS 7·5 points [95% CI 5·3 to 9·8]; p<0·0001; mean between-group difference in bodyweight at week 52 -8·4% [-9·2 to -7·5]; p<0·0001). For the confirmatory secondary endpoints, 6-min walk distance (mean between-group difference at week 52 17·1 metres [9·2 to 25·0]) and the hierarchical composite endpoint (win ratio 1·65 [1·42 to 1·91]) were significantly improved, and CRP concentrations (treatment ratio 0·64 [0·56 to 0·72]) were significantly reduced, in the semaglutide group compared with the placebo group (p<0·0001 for all comparisons). For the dual primary endpoints, the efficacy of semaglutide was largely consistent across multiple subgroups, including those defined by age, race, sex, BMI, systolic blood pressure, baseline CRP, and left ventricular ejection fraction. 161 serious adverse events were reported in the semaglutide group compared with 301 in the placebo group. In this prespecified pooled analysis of the STEP-HFpEF and STEP-HFpEF DM trials, semaglutide was superior to placebo in improving heart failure-related symptoms and physical limitations, and reducing bodyweight in participants with obesity-related heart failure with preserved ejection fraction. These effects were largely consistent across patient demographic and clinical characteristics. Semaglutide was well tolerated. Novo Nordisk.

Spending response to cash transfers to shield households from inflation: Evidence from bank accounts

We use daily data from a large Spanish financial institution to analyze the spending responses to a one-off cash transfer to mitigate the impact of the surge in inflation during the 2022–2023 period. Using a staggered difference-in-differences estimator robust to treatment effect heterogeneity, we find that most of the expenditure is concentrated around the date of the transfer and that the average marginal propensity to spend is around 30%. The results are robust to changes in the main specification.

인과 포레스트를 활용한 대졸 청년층의 직무와 전공 일치가 첫 직장 만족도에 미치는 효과 분석

This study was conducted to explore the impact of major-job matching on first job satisfaction&#x0D; among college graduates. To do so, the casual forest technique, a machine learning-based causal&#x0D; inference method, was applied to the Graduates Occupational Mobility Survey (GOMS) 2019 data to&#x0D; analyze the average treatment effects of major-job matching on first job satisfaction and the&#x0D; heterogeneous treatments effects by covariates. The main findings are as follows. First, major-job&#x0D; matching had a positive and significant effect on first job satisfaction even after controlling for&#x0D; personal, university, and workplace characteristics as covariates. Second, heterogeneous treatment effects&#x0D; were observed by gender, participation in job camps, and business type. Based on the key findings,&#x0D; the implications for high job satisfaction among young college graduates through employment related&#x0D; to their majors were discussed.

다중비교 인과포레스트를 활용한 대학입학전형에 따른 대학생활 성과 분석

Over the years, university admission types have primarily been the subject of social discussions on the&#x0D; dimension of fairness. However, considering the original purpose of university admissions system, it is also&#x0D; crucial to evaluate how well each type performs in selecting students who can successfully navigate academic&#x0D; life at each university. Accordingly, this study aimed to analyze the university life outcomes (sense of&#x0D; belonging, adaptability, intention to reselect the university, intention to reselect the major, GPA) according&#x0D; to different university admissions types: the admission officer(AO) system, high school grade-focused(HSG)&#x0D; selection system, and College Scholastic Ability Test-focused(CSAT) selection system, using the data from the&#x0D; Korea Education Longitudinal Study(KELS) 2013 collected in ’20~’21(N=2,011). Assuming the effects may&#x0D; vary according to the baseline characteristics, the study employs the Conditional Average Treatment Effects&#x0D; (CATE) perspective through the Multi-arm Causal Forest technique(Athey, Tibshirani, &amp; Wager, 2019). The&#x0D; main findings are summarized as follows: First, on average, students admitted through AO system exhibit&#x0D; superior outcomes across all considered aspects of college life, with a notable emphasis on higher intentions&#x0D; to reselect both the university and major. Second, it appears that parental income and education, high&#x0D; school academic achievement levels and their university-level average were significant conditioning variables&#x0D; predicting heterogeneous treatment effects on college life outcomes. Last, the effect heterogeneity of CSAT&#x0D; system, depending on the values of major conditioning variables, was more prominent compared to that of&#x0D; AO system. Based on these findings, suggestions for interpreting the research results and directions for&#x0D; future studies were provided.

Responsibility for impairment shapes the perceived deservingness of welfare claimants with disabilities

AbstractWhen do people support government assistance for people with disabilities? Disability welfare programs account for large shares of national welfare budgets, but little is known about public attitudes toward disabled welfare claimants. Drawing on psychological research in stereotype content, we argue that attitudes toward welfare for people with disabilities are likely to be more conditional than previously acknowledged. In two nationally representative, preregistered survey experiments in Wales (N = 3393) and Scotland (1707), we ask respondents to evaluate the deservingness of a fictitious disabled claimant to government assistance. We manipulate the claimant's outgroup status and the manner in which they acquired their impairment. We find that disabled claimants perceived as even somewhat responsible for their impairments are considered substantially less deserving of government assistance than those perceived not responsible, even when their needs for assistance are identical. Contrary to expectations, we find relatively modest and inconsistent outgroup penalties in perceived deservingness. Finally, we find large heterogeneous treatment effects among respondents holding to more authoritarian social values. These results challenge conventional wisdom regarding the universality of support for disability welfare and help explain why voters may not be inclined to punish politicians who propose cuts to programs for even stereotypically high‐deserving groups.

Individualized Treatment Effect Prediction with Machine Learning — Salient Considerations

BackgroundMachine learning–based approaches that seek to accomplish individualized treatment effect prediction have gained traction; however, some salient challenges lack wider recognition.MethodsWe describe key methodologic considerations for individualized treatment effect prediction models using data from the Treatment of Preserved Cardiac Function Heart Failure with an Aldosterone Antagonist Trial for spironolactone in heart failure with preserved ejection fraction. The causal survival forest algorithm was used for model development. Calibration and discrimination were evaluated using a bootstrapping-based internal validation procedure. Observed benefits were described for predicted benefit quartiles and quartiles of a known effect modifier: ejection fraction. A negative control analysis with noncardiovascular death as the outcome was implemented to detect confounding.ResultsAmong 3445 participants, 671 events occurred over a median of 3.3 years of follow-up. In internal validation, a higher average observed benefit was noted among patients in the highest quartile of predicted benefit. The median (interquartile range) of the observed restricted mean survival time difference at 3.3 years at the highest quartile of model-predicted benefit was 62 days (32 to 83) and was 47 days (26 to 67) at the lowest quartile of ejection fraction. Body-mass index had higher contribution to prediction of benefit relative to other included measures (33.7% vs. glomerular filtration rate [27.3%], ejection fraction [15.1%], and younger age [12.8%]) No benefit was observed for noncardiovascular death at higher model-predicted benefit quartiles, although benefit for noncardiovascular death was observed at lower quartiles.ConclusionsCarefully applied and validated predictive models hold promise in identifying heterogeneous treatment effects and are useful for hypothesis generation regarding the role of phenotypic characteristics in modifying the benefit of experimental interventions in clinical trials. (Funded by the National Heart, Lung, and Blood Institute; ClinicalTrials.gov number, NCT00094302.)

Uncertainty Quantification in Heterogeneous Treatment Effect Estimation with Gaussian-Process-Based Partially Linear Model

Estimating heterogeneous treatment effects across individuals has attracted growing attention as a statistical tool for performing critical decision-making. We propose a Bayesian inference framework that quantifies the uncertainty in treatment effect estimation to support decision-making in a relatively small sample size setting. Our proposed model places Gaussian process priors on the nonparametric components of a semiparametric model called a partially linear model. This model formulation has three advantages. First, we can analytically compute the posterior distribution of a treatment effect without relying on the computationally demanding posterior approximation. Second, we can guarantee that the posterior distribution concentrates around the true one as the sample size goes to infinity. Third, we can incorporate prior knowledge about a treatment effect into the prior distribution, improving the estimation efficiency. Our experimental results show that even in the small sample size setting, our method can accurately estimate the heterogeneous treatment effects and effectively quantify its estimation uncertainty.

Abstract 6467: Causal personalized treatment estimation framework using real-world electronic health record (EHR) data to inform cancer care decisions

Abstract Background: The average treatment effects (ATEs) from randomized controlled trials often do not generalize across populations, or they are not sufficiently personalized to capture the benefits and risks to individual patients. Real-world data aids in validating and extending trial results, but can introduce biases that invalidate the causal interpretation. Here, we propose a robust statistical framework, which uses real-world EHR data to evaluate the efficacy of cancer treatments at both population and individual levels. We demonstrate the ability of our framework to deliver personalized insights in metastatic prostate cancer treatments. Methods: We assembled a retrospective cohort of metastatic prostate cancer patients from a large academic medical center. We gathered demographic, comorbidity, cancer treatment, and outcome data from the EHR or natural language processing of clinician narratives, and social determinants of health from Census data. We used doubly robust machine learning with bootstrap standard errors to compute the ATE and personalized treatment estimates (PTEs), comparing the impact of ADT (androgen depravation therapy) + chemotherapy (ACT) and ADT + combination anti-androgen therapy (ACAT) on percentage reduction in PSA levels in metastatic prostate cancer. Results: The baseline characteristics of our cohort are summarized in Table 1a. Our approach found that ACAT reduces PSA levels significantly more than ACT, and captured the heterogeneity in treatment effects across individuals (Table 1b). Through simulation studies, we demonstrated that any reasonable bias from unobserved confounding or missing information was small compared to the estimated treatment effects and their standard errors. Conclusion: We presented a doubly robust framework and demonstrated its utility in analyzing real-world data, enabling us to uncover the diversity in treatment responses among individuals with metastatic prostate cancer. Results summary a. Baseline demographic characteristics. ADT + chemotherapy N = 302 ADT + combination therapy N = 114 Treatments Any ADT + Docetaxel or Cabazitaxel Any ADT + Abiraterone or apalutamide or Enzalutamide Age Mean (SD) 69.3 (8.76) 66.7 (8.17) Median [Min, Max] 69.0 [47.0, 94.0] 66.5 [46.0, 86.0] Race Asian 8 (2.6%) 2 (1.8%) Black or African American 48 (15.9%) 7 (6.1%) White or Caucasian 224 (74.2%) 99 (86.8%) Other 22 (7.3%) 6 (5.3%) Max PSA treatment start (ng/mL) Mean (SD) 34.9 (98.5) 83.3 (448) b. Treatment effects estimation ATE (95% CI) 32.0% [24.6%, 40.8%] PTEs: 25th and 75th percentiles 29.0%, 34.3% Citation Format: Sumana Srivatsa, David Walsh, Mayada Aljehani, Jason Weinreb, Nathan Becker, Benjamin H. Ellis, Renee George, Xingyao Chen, Naim Matasci, Reva Basho, Mitchell E. Gross, Christine L. Swisher. Causal personalized treatment estimation framework using real-world electronic health record (EHR) data to inform cancer care decisions [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 6467.

Survival, immune response and growth of Penaeid shrimp as affected by immunostimulants: A meta-analysis

Immunostimulants represent the most innovative approach for combating shrimp diseases. They are molecules that effectively enhance the host's nonspecific defenses against invading microorganisms. However, methodological differences exist among immunostimulants based on the same source. Therefore, conducting a meta-analysis is essential to derive valid conclusions. The effect size value utilized in this study was Hedges' d. Heterogeneity among studies was assessed using the DerSimonian and Laird tests (Q-statistic). Meta-regression analysis was conducted to explore the sources of heterogeneity in treatment effects. In this study, dose served as a covariate because it was the only continuous variable that significantly contributed to the observed heterogeneity. Funnel plots and the fail-safe number were employed to assess publication bias within the datasets. The article collection process followed the PRISMA methodology. Based on the results of the meta-analysis and meta-regression conducted with 83 articles, it can be concluded that immunostimulants have a significant effect, characterized by high category standard mean difference (SMD) values, on the survival, growth, and immune response of Penaeid family shrimp. Among potential immunostimulants options, algae ingredients exhibited the most favorable effects on the survival, growth, and immune response of Penaeid family shrimp. Subgroup analysis outcomes revealed that various extraction methods significantly impacted the efficacy of immunostimulants, with the ethanol solvent method proving to be the most effective. Among different administration methods, no significant effect was observed on immunostimulant efficacy across all parameters, with positive SMD values for all administration methods. Regarding challenged test pathogens, immunostimulants were observed to enhance immune response, survival, and weight gain against various pathogens. Meta-regression results indicated that algal treatments had a lower optimal dose point, leading to decreased efficacy as the dose increased. In contrast, fungi exhibited a higher optimum dose point, resulting in increased efficacy at higher doses.

Comparing type 1 and type 2 error rates of different tests for heterogeneous treatment effects.

Psychologists are increasingly interested in whether treatment effects vary in randomized controlled trials. A number of tests have been proposed in the causal inference literature to test for such heterogeneity, which differ in the sample statistic they use (either using the variance terms of the experimental and control group, their empirical distribution functions, or specific quantiles), and in whether they make distributional assumptions or are based on a Fisher randomization procedure. In this manuscript, we present the results of a simulation study in which we examine the performance of the different tests while varying the amount of treatment effect heterogeneity, the type of underlying distribution, the sample size, and whether an additional covariate is considered. Altogether, our results suggest that researchers should use a randomization test to optimally control for type 1 errors. Furthermore, all tests studied are associated with low power in case of small and moderate samples even when the heterogeneity of the treatment effect is substantial. This suggests that current tests for treatment effect heterogeneity require much larger samples than those collected in current research.

Finerenone cardiovascular and kidney outcomes by age and sex: FIDELITY post hoc analysis of two phase 3, multicentre, double-blind trials

ObjectivesThis study aimed to evaluate the efficacy and safety of finerenone, a selective, non-steroidal mineralocorticoid receptor antagonist, on cardiovascular and kidney outcomes by age and/or sex.DesignFIDELITY post hoc analysis; median...

Combining randomized and non-randomized data to predict heterogeneous effects of competing treatments.

Some patients benefit from a treatment while others may do so less or do not benefit at all. We have previously developed a two-stage network meta-regression prediction model that synthesized randomized trials and evaluates how treatment effects vary across patient characteristics. In this article, we extended this model to combine different sources of types in different formats: aggregate data (AD) and individual participant data (IPD) from randomized and non-randomized evidence. In the first stage, a prognostic model is developed to predict the baseline risk of the outcome using a large cohort study. In the second stage, we recalibrated this prognostic model to improve our predictions for patients enrolled in randomized trials. In the third stage, we used the baseline risk as effect modifier in a network meta-regression model combining AD, IPD randomized clinical trial to estimate heterogeneous treatment effects. We illustrated the approach in the re-analysis of a network of studies comparing three drugs for relapsing-remitting multiple sclerosis. Several patient characteristics influence the baseline risk of relapse, which in turn modifies the effect of the drugs. The proposed model makes personalized predictions for health outcomes under several treatment options and encompasses all relevant randomized and non-randomized evidence.

Contrast-specific propensity scores for causal inference with multipleinterventions.

Existing methods that use propensity scores for heterogeneous treatment effect estimation on non-experimental data do not readily extend to the case of more than two treatment options. In this work, we develop a new propensity score-based method for heterogeneous treatment effect estimation when there are three or more treatment options, and prove that it generates unbiased estimates. We demonstrate our method on a real patient registry of patients in Singapore with diabetic dyslipidemia. On this dataset, our method generates heterogeneous treatment recommendations for patients among three options: Statins, fibrates, and non-pharmacological treatment to control patients' lipid ratios (total cholesterol divided by high-density lipoprotein level). In our numerical study, our proposed method generated more stable estimates compared to a benchmark method based on a multi-dimensional propensity score.